Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bitset: add range counting and parallel bit extract/deposit operations #184

Merged
merged 2 commits into from
Dec 9, 2024

Conversation

tsenart
Copy link
Contributor

@tsenart tsenart commented Dec 4, 2024

Add three new operations to BitSet:

  • OnesBetween(from, to uint) counts set bits in range [from, to)
  • Extract/ExtractTo compacts bits from positions specified by a mask
  • Deposit/DepositTo spreads bits to positions specified by a mask

OnesBetween operates at the word level where possible, handling edge words with masks. It returns 0 for invalid ranges where from >= to.

Extract takes bits from positions where mask is set and packs them into consecutive positions starting at 0. For example, if mask has bits set at positions 1,4,5 then Extract will take bits at those positions from the source and pack them into positions 0,1,2.

Deposit is the inverse of Extract - it spreads bits from consecutive positions into positions specified by mask.

The parallel bit operations are implemented using byte-wise lookup tables rather than CPU-specific instructions like PEXT/PDEP, making them portable while maintaining good performance.

Added extensive testing:

  • Property-based tests with 1M iterations verifying correctness
  • Edge cases like single words, cross-word boundaries, empty sets
  • Invariant checking (e.g. result bits ⊆ mask bits)
  • Benchmarks across sizes from 64 to 65536 bits

These operations are useful for:

  • Efficient bit field compression/decompression
  • Sparse set operations
  • Bit permutation tasks

The implementation aims to be both correct and fast, operating at the word level where possible while properly handling edges.

Benchmarks

goos: darwin
goarch: arm64
pkg: github.com/bits-and-blooms/bitset
cpu: Apple M3 Max
                                                │    bench    │
                                                │   sec/op    │
BitSetOnesBetween/size=64/density=0.1-16          3.026n ± 1%
BitSetOnesBetween/size=64/density=0.5-16          2.959n ± 3%
BitSetOnesBetween/size=64/density=0.9-16          3.058n ± 1%
BitSetOnesBetween/size=256/density=0.1-16         3.154n ± 2%
BitSetOnesBetween/size=256/density=0.5-16         3.179n ± 1%
BitSetOnesBetween/size=256/density=0.9-16         3.146n ± 1%
BitSetOnesBetween/size=1024/density=0.1-16        4.938n ± 1%
BitSetOnesBetween/size=1024/density=0.5-16        4.934n ± 1%
BitSetOnesBetween/size=1024/density=0.9-16        4.881n ± 1%
BitSetOnesBetween/size=4096/density=0.1-16        8.661n ± 1%
BitSetOnesBetween/size=4096/density=0.5-16        8.787n ± 2%
BitSetOnesBetween/size=4096/density=0.9-16        8.582n ± 1%
BitSetOnesBetween/size=16384/density=0.1-16       29.05n ± 0%
BitSetOnesBetween/size=16384/density=0.5-16       30.11n ± 0%
BitSetOnesBetween/size=16384/density=0.9-16       29.34n ± 0%
PEXT-16                                           3.243n ± 2%
PDEP-16                                           3.219n ± 1%
BitSetExtractDeposit/size=64/fn=ExtractTo-16      10.55n ± 0%
BitSetExtractDeposit/size=64/fn=DepositTo-16      9.812n ± 1%
BitSetExtractDeposit/size=256/fn=ExtractTo-16     35.30n ± 1%
BitSetExtractDeposit/size=256/fn=DepositTo-16     38.16n ± 1%
BitSetExtractDeposit/size=1024/fn=ExtractTo-16    134.5n ± 1%
BitSetExtractDeposit/size=1024/fn=DepositTo-16    157.2n ± 1%
BitSetExtractDeposit/size=4096/fn=ExtractTo-16    580.1n ± 2%
BitSetExtractDeposit/size=4096/fn=DepositTo-16    642.0n ± 1%
BitSetExtractDeposit/size=16384/fn=ExtractTo-16   2.412µ ± 4%
BitSetExtractDeposit/size=16384/fn=DepositTo-16   2.459µ ± 2%
BitSetExtractDeposit/size=65536/fn=ExtractTo-16   10.12µ ± 1%
BitSetExtractDeposit/size=65536/fn=DepositTo-16   9.832µ ± 1%
geomean                                           30.55n

Add three new operations to BitSet:

- OnesBetween(from, to uint) counts set bits in range [from, to)
- Extract/ExtractTo compacts bits from positions specified by a mask
- Deposit/DepositTo spreads bits to positions specified by a mask

OnesBetween operates at the word level where possible, handling edge
words with masks. It returns 0 for invalid ranges where from >= to.

Extract takes bits from positions where mask is set and packs them into
consecutive positions starting at 0. For example, if mask has bits set
at positions 1,4,5 then Extract will take bits at those positions from
the source and pack them into positions 0,1,2.

Deposit is the inverse of Extract - it spreads bits from consecutive
positions into positions specified by mask.

The parallel bit operations are implemented using byte-wise lookup
tables rather than CPU-specific instructions like PEXT/PDEP, making
them portable while maintaining good performance.

Added extensive testing:
- Property-based tests with 1M iterations verifying correctness
- Edge cases like single words, cross-word boundaries, empty sets
- Invariant checking (e.g. result bits ⊆ mask bits)
- Benchmarks across sizes from 64 to 65536 bits

These operations are useful for:
- Efficient bit field compression/decompression
- Sparse set operations
- Bit permutation tasks

The implementation aims to be both correct and fast, operating at
the word level where possible while properly handling edges.

Benchmarks

```
goos: darwin
goarch: arm64
pkg: github.com/bits-and-blooms/bitset
cpu: Apple M3 Max
                                                │    bench    │
                                                │   sec/op    │
BitSetOnesBetween/size=64/density=0.1-16          3.026n ± 1%
BitSetOnesBetween/size=64/density=0.5-16          2.959n ± 3%
BitSetOnesBetween/size=64/density=0.9-16          3.058n ± 1%
BitSetOnesBetween/size=256/density=0.1-16         3.154n ± 2%
BitSetOnesBetween/size=256/density=0.5-16         3.179n ± 1%
BitSetOnesBetween/size=256/density=0.9-16         3.146n ± 1%
BitSetOnesBetween/size=1024/density=0.1-16        4.938n ± 1%
BitSetOnesBetween/size=1024/density=0.5-16        4.934n ± 1%
BitSetOnesBetween/size=1024/density=0.9-16        4.881n ± 1%
BitSetOnesBetween/size=4096/density=0.1-16        8.661n ± 1%
BitSetOnesBetween/size=4096/density=0.5-16        8.787n ± 2%
BitSetOnesBetween/size=4096/density=0.9-16        8.582n ± 1%
BitSetOnesBetween/size=16384/density=0.1-16       29.05n ± 0%
BitSetOnesBetween/size=16384/density=0.5-16       30.11n ± 0%
BitSetOnesBetween/size=16384/density=0.9-16       29.34n ± 0%
PEXT-16                                           3.243n ± 2%
PDEP-16                                           3.219n ± 1%
BitSetExtractDeposit/size=64/fn=ExtractTo-16      10.55n ± 0%
BitSetExtractDeposit/size=64/fn=DepositTo-16      9.812n ± 1%
BitSetExtractDeposit/size=256/fn=ExtractTo-16     35.30n ± 1%
BitSetExtractDeposit/size=256/fn=DepositTo-16     38.16n ± 1%
BitSetExtractDeposit/size=1024/fn=ExtractTo-16    134.5n ± 1%
BitSetExtractDeposit/size=1024/fn=DepositTo-16    157.2n ± 1%
BitSetExtractDeposit/size=4096/fn=ExtractTo-16    580.1n ± 2%
BitSetExtractDeposit/size=4096/fn=DepositTo-16    642.0n ± 1%
BitSetExtractDeposit/size=16384/fn=ExtractTo-16   2.412µ ± 4%
BitSetExtractDeposit/size=16384/fn=DepositTo-16   2.459µ ± 2%
BitSetExtractDeposit/size=65536/fn=ExtractTo-16   10.12µ ± 1%
BitSetExtractDeposit/size=65536/fn=DepositTo-16   9.832µ ± 1%
geomean                                           30.55n
```
@tsenart tsenart force-pushed the ext-dep-ones-between branch from 03e3172 to 5a7da69 Compare December 4, 2024 01:03
@lemire lemire merged commit 2108e73 into bits-and-blooms:master Dec 9, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants